Real-time operator control of robots has been accomplished with electro-mechanical controls such as joysticks and multiple axis hand grips. These devices suffer from a limited range of motion due to being constrained by the geometry of the control device. In other applications, such as surgery, the operator's hand and finger motions used to operate the device do not closely approximate those motions he would use in conducting the operation by hand. This requires the surgeon to use a different repertoire of hand motions for the robot control, than he would for conducting the operation by hand. Other devices such as a glove actuator, while more closely approximating the actual motion of the hand, suffers from a lack of accuracy regarding the motion of the instrument the hand and fingers grasp, and it is the working end of the instrument which is being mimicked by the robot's tools that do the work. Other interfaces have been developed that rely on multiple cameras to record the motion of the operator's hands with or without faux instruments, but these can also suffer from a lack of accuracy.
These devices also suffer from mechanical wear and tear, which compromises accuracy and require maintenance.
These devices suffer from latency, especially when the operator is separated from the worksite by sufficient distances that there is a significant delay in transmission.
It is an object of some aspects of the invention to address one or more of the above existing concerns. Other concerns may also be addressed in those aspects, or separately in other aspects of the invention as will be evident from the remainder of this specification.
In a first aspect, the invention provides a method comprising the steps of actively generating an image pattern on a surface of a first object, detecting the image pattern on the surface of the first object, wherein either the step of actively generating or the step of detecting is performed at a second object spaced away from the first object, and determining parameters of the relative poses of the second object and the surface utilizing the detected image pattern and utilizing reference data for actively generating the image pattern.
The method may further comprise the step of actively displaying on the first surface an image of a remote process that is controlled in accordance with the determined parameters of the pose of the second object.
The step of actively generating may comprise the step of projecting a known image pattern to actively generate the image pattern on the surface of the first object, wherein the step of projecting is from either the second object if the step of actively generating is performed at the second object or a first location other than the second object and the first object if the step of detecting is performed at the second object.
The step of projecting may comprise projecting the image pattern from the second object. The step of detecting may comprise detecting the image pattern at the surface of the first object. The step of projecting may comprise projecting the image pattern from the first location. The step of detecting further comprises detecting the image pattern from a second location other than the first object and the second object.
The method may further comprise the step of maintaining the first object in a known pose during the steps of projecting and detecting. The method may further comprise the step of maintaining the second object in a known pose during the steps of projecting and detecting.
The surface of the first object may be substantially planar.
The method may further comprise the step of detecting movement of the detected pattern, and the step of determining parameters of the pose of the second object comprises determining movement of parameters of the pose of the second object from the detected movement of the detected pattern.
The method may further comprise the step of detecting linear movement of the second object parallel to the surface by detecting motion against texturing on the surface.
The step of projecting may further comprise projecting the image pattern such that the image pattern is asymmetrical about an axis of rotation in line with a direction of projection of the image pattern. The step of projecting may further comprise projecting the image pattern such that the size of the image pattern varies continuously with distance from the first object in line with a direction of projection of the image pattern.
The step of actively generating the image pattern may include actively generating elements of the image pattern over time, and the step of detecting includes detecting elements of the formed image pattern in synchronization with actively generating the image elements.
The steps of actively generating and detecting may comprise actively generating on the surface which surface forms a three dimensional cavity with access for the second object through un opening in the first object, and detecting the image pattern formed on such surface, respectively.
The surface may comprise a plurality of substantially planar sub-surfaces. The step of projecting further comprises projecting the image pattern as a combination of three or more spot beams of known relationship. The step of actively generating may further comprise actively generating the image pattern as a combination of three or more spot beams of known relationship.
The step of projecting may comprise projecting the image pattern with image pattern elements directed at a plurality of angles about an axis of the second object. The method may further comprise the step of user imparting movement of the second object.
The step of projecting may further comprise projecting encoded information, other than pose-related information, in an image pattern projected from the second object.
The step of determining an element of the pose of the second object may further comprise determining a distance from the image pattern on the surface of the first object to a reference point on the second object based upon the size of the detected image pattern.
In a second aspect the invention provides a method of controlling instruments of a surgical robot in use on a heart, the method comprising the steps of receiving a signal that a heart is about contract, and initiating movement of the surgical robot instruments so as to accommodate movement of the heart in the vicinity of the instruments during contraction as movement of the heart occurs.
The step of receiving may further comprise receiving a signal related to an anticipated nature of the contraction, and the step of initiating further comprises utilizing the anticipated nature of the contraction from the signal to control the accommodation. The method may comprise the steps of detecting a contour of movement of a heart by, projecting an image pattern on to a surface of the heart in the vicinity of the instrument, repeatedly detecting the image pattern formed on the surface of the heart, and determining movement of the heart based on a transformation of the detected image pattern from reference image pattern data, and moving the surgical robot instruments so as to accommodate the contour of movement of the heart in the vicinity of the instrument, so that operator intended motions can be carried out from this normalized position.
The third aspect the invention provides a method of controlling an instrument of a surgical robot comprising the steps of detecting a contour of movement of a heart by, projecting an image pattern on to a surface of the heart in the vicinity of the instrument, repeatedly detecting the image pattern formed on the surface of the heart, and determining movement of the heart based on a transformation of the detected image pattern from reference image pattern data, and moving the surgical robot instruments so as to accommodate the contour of movement of the heart in the vicinity of the instrument, so that operator intended motions can be carried out from this normalized position.
In a fourth aspect the invention provides a robot system comprising a robot including and controlling an instrument, controls for an operator to control the robot to operate the instrument, a controller for determining quantified information related to motion of the instrument, and a display for displaying the information from the controller to an operator of the robot during use of the robot.
In a fifth aspect the invention provides a method of conveying information regarding the latency between motion of a controller and motion of an instrument in a remote process controlled by the controller, the method comprising displaying to an operator of the controller an image of the instrument and at least a portion of the remote process surrounding the instrument in a direction of motion of the instrument, and overlaying on the displayed image, an image of the instrument in a pose requested by motion of the controller, such that the operator can see an image of the actual pose of the instrument, and the requested pose of the instrument.
In a sixth aspect the invention provides a method of conveying information regarding the latency between motion of a controller of a surgical robot and motion of an instrument of the surgical robot controlled by the controller, the method comprising displaying on a display visible to an operator of the controller, an image of the instrument and at least a portion of a surgical field surrounding the instrument in a direction of motion or the instrument, and overlaying on the displayed image, an image of the instrument in a pose requested by motion of the controller, such that the operator can see an image of the actual pose of the instrument, and the requested pose of the instrument.
In a seventh aspect the invention provides a method of controlling latency between motion of a controller and motion of the instrument in a remote process controlled by the controller, the method comprising the steps of acquiring an original image of the instrument and at least a portion of a surgical field surrounding the instrument in a direction of motion of the instrument, and displaying the original image to an operator of the controller, acquiring an instruction from the controller to move the instrument to an instructed pose relative to the original image, transmitting the instruction and information to identify the original image to the remote process, acquiring an updated image of the remote process, performing pattern recognition at the remote process on the image identified by the transmitted information and the updated image to determine a desired pose of the instrument relative to the updated image that corresponds to the instructed pose on the original image, and moving the instrument to the desired pose.
In an eighth aspect the invention provides a method comprising the steps of actively displaying on a surface of a first object an image of a remote process that is controlled in accordance with parameters of the pose of a second object spaced away from the first object, detecting an image pattern on the surface of the first object, wherein either the image pattern is actively generated from the second object or the image pattern is detected at the second object, determining parameters of the relative poses of the second object and the surface utilizing the detected image pattern and utilizing reference data for the image pattern, and controlling the remote process in accordance with the determined parameters at the pose of the second object.
In a ninth aspect the invention provides a method comprising the steps of projecting a known image pattern on to a surface of a first object, wherein the step of projecting is from either a second object or a first location other than the second object and the first object, and the first object, second object and first location are at a distance from one another, detecting the image pattern formed on the surface of the first object, wherein if the step of projecting is from the second object then the step of detecting is from either the first object, second object or a second location other than the first and the second object, and if the step of projecting is from the first location then the step of detecting is from the second object, and determining parameters of the pose of the second object utilizing the detected image pattern and reference image pattern data for the known pattern.
In a tenth aspect the invention provides a method of controlling an instrument of a robot comprising the steps of detecting a contour of movement of an object being worked by the instrument, projecting an image pattern on to a surface of the object in the vicinity of the instrument, repeatedly detecting the image pattern formed on the surface of the object, and determining movement of the object based on a transformation of the detected image pattern from reference image pattern data, and moving the robot instruments so as to accommodate the contour of movement of the object in the vicinity of the instrument, so that operator intended motions can be carried out from this normalized position.
In a eleventh aspect the invention provides an input interface comprising a pattern generator for actively generating an image pattern on a surface of a first object, a detector for detecting the image pattern on the surface of the first object, wherein the pattern generator or the detector is at a second object spaced away from the first object, and a computer for determining parameters of the relative poses of the second object and the surface utilizing the detected image pattern from the detector and utilizing reference data for actively generating the image pattern.
In a twelfth aspect the invention provides a system comprising a surgical robot including an instrument controlled by the robot, a computer for receiving a signal that a heart being operated on by the instrument is about to contract, and generating instructions to the robot to initiate movement of the surgical robot instrument so as to accommodate movement of the heart in the vicinity of the instruments during contraction as movement of the heart occurs.
In a thirteenth aspect the invention provides a robot system comprising a robot including and controlling an instrument, controls for an operator to control the robot to operate the instrument, a controller for determining quantified information related to motion of the instrument, and a display for displaying the information from the controller to an operator of the robot during use of the robot.
In a fourteenth aspect the invention provides a system for conveying information regarding the latency between motion of a controller and motion of an instrument in a remote process controlled by the controller, the system comprising a computer and a display for displaying to an operator of the controller an image of the instrument and at least a portion of the remote process surrounding the instrument in a direction of motion of the instrument, and an overlay on the displayed image, an image of the instrument in a pose requested by motion of the controller, such that the operator can see an image of the actual pose of the instrument, and the requested pose of the instrument.
In a fifteenth aspect the invention provides system for conveying information regarding the latency between motion of a controller of a surgical robot and motion of an instrument of the surgical robot controlled by the controller, the system comprising a computer and a display for displaying on a display visible to an operator of the controller an image of the instrument and at least a portion of a surgical field surrounding the instrument in a direction of motion of the instrument, and overlaying on the displayed image, an image of the instrument in a pose requested by motion of the controller, such that the operator can see an image of the actual pose of the instrument, and the requested pose of the instrument.
In a sixteenth aspect the invention provides a system for controlling latency between motion of a controller and motion of the instrument in a remote process controlled by the controller, the system comprising a camera for acquiring an original image of the instrument and at least a portion of a surgical field surrounding the instrument in a direction of motion of the instrument, and a display for displaying the original image to an operator of the controller, acquiring an instruction from the controller to move the instrument to an instructed pose relative to the original image, and transmitting the instruction and information to identify the original image to the remote process, wherein the camera is also for acquiring an updated image of the remote process a computer for performing pattern recognition at the remote process on the image identified by the transmitted information and the updated image to determine a desired pose of the instrument relative to the updated image that corresponds to the instructed pose on the original image, and instructing the remote process to move the instrument to the desired pose.
In a seventeenth aspect, the invention provides a computer readable medium storing program instructions executable by one or more processors in one or more computers for causing the computers to implement the method of any one of the method aspects.
Other aspects of the present invention and detailed additional features of the above aspects will be evident based upon the detailed description, FIGS. and claims herein, including for example systems corresponding to the methods of the above aspects, methods corresponding to the systems of the above aspects, input interfaces, wands, robots, computing systems, alignment systems, software, methods of using the above, and the like.